Cross-Modal Mutual Learning for Audio-Visual Speech Recognition and Manipulation

As a key characteristic in audio-visual speech recognition (AVSR), relating linguistic information observed across visual and audio data has been challenge, benefiting not only audio/visual (ASR/VSR) but also for manipulating within/across modalities. In this paper, we present feature disentanglement-based framework jointly addressing the above tasks. By advancing cross-modal mutual learning strategies, our model is able to convert or audio-based features into modality-agnostic representations. Such derived representations allow one perform ASR, VSR, AVSR, manipulate output based on desirable subject identity content information. We extensive experiments different synthesis tasks show that performs favorably against state-of-the-art approaches each individual task, while ours unified solution tackle aforementioned